<!DOCTYPE html>
<html>
<head>
    

    

    



    <meta charset="utf-8">
    
    
    
    <title>Scrapy学习（四） 爬取微博数据 | 四畳半神话大系 | 圈地自萌</title>
    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
    
    <meta name="theme-color" content="#3F51B5">
    
    
    <meta name="keywords" content="学习,Scrapy">
    <meta name="description" content="前言
Scrapy学习（三） 爬取豆瓣图书信息

接上篇之后。这次来爬取需要登录才能访问的微博。爬虫目标是获取用户的微博数、关注数、粉丝数。为建立用户关系图(尚未实现)做数据储备">
<meta property="og:type" content="article">
<meta property="og:title" content="Scrapy学习（四） 爬取微博数据">
<meta property="og:url" content="http://amoyiki.com/2017/02/13/Scrapy学习（四）-爬取微博数据/index.html">
<meta property="og:site_name" content="四畳半神话大系">
<meta property="og:description" content="前言
Scrapy学习（三） 爬取豆瓣图书信息

接上篇之后。这次来爬取需要登录才能访问的微博。爬虫目标是获取用户的微博数、关注数、粉丝数。为建立用户关系图(尚未实现)做数据储备">
<meta property="og:image" content="http://oi6538cys.bkt.clouddn.com/snipaste20170213_132630.png">
<meta property="og:image" content="http://oi6538cys.bkt.clouddn.com/snipaste20170213_141218.png">
<meta property="og:updated_time" content="2017-02-13T06:50:38.033Z">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Scrapy学习（四） 爬取微博数据">
<meta name="twitter:description" content="前言
Scrapy学习（三） 爬取豆瓣图书信息

接上篇之后。这次来爬取需要登录才能访问的微博。爬虫目标是获取用户的微博数、关注数、粉丝数。为建立用户关系图(尚未实现)做数据储备">
<meta name="twitter:image" content="http://oi6538cys.bkt.clouddn.com/snipaste20170213_132630.png">
    
        <link rel="alternative" href="/atom.xml" title="四畳半神话大系" type="application/atom+xml">
    
    <link rel="shortcut icon" href="/favicon.ico">
    <link rel="stylesheet" href="/css/style.css?v=1.4.14">
    <script>window.lazyScripts=[]</script>
</head>

<body>
    <div id="loading" class="active"></div>

    <aside id="menu" class="hide" >
  <div class="inner flex-row-vertical">
    <a href="javascript:;" class="header-icon waves-effect waves-circle waves-light" id="menu-off">
        <i class="icon icon-lg icon-close"></i>
    </a>
    <div class="brand-wrap">
      <div class="brand">
        <a href="/" class="avatar waves-effect waves-circle waves-light">
          <img src="/images/avatar.jpg">
        </a>
        <hgroup class="introduce">
          <h5 class="nickname">amoyiki</h5>
          <a href="mailto:amoyiki#gmail.com" title="amoyiki#gmail.com" class="mail">amoyiki#gmail.com</a>
        </hgroup>
      </div>
    </div>
    <div class="scroll-wrap flex-col">
      <ul class="nav">
        
            <li class="waves-block waves-effect">
              <a href="/"  >
                <i class="icon icon-lg icon-home"></i>
                主页
              </a>
            </li>
        
            <li class="waves-block waves-effect">
              <a href="/archives"  >
                <i class="icon icon-lg icon-archives"></i>
                Archives
              </a>
            </li>
        
            <li class="waves-block waves-effect">
              <a href="/tags"  >
                <i class="icon icon-lg icon-tags"></i>
                Tags
              </a>
            </li>
        
            <li class="waves-block waves-effect">
              <a href="/categories"  >
                <i class="icon icon-lg icon-th-list"></i>
                Categories
              </a>
            </li>
        
            <li class="waves-block waves-effect">
              <a href="https://github.com/amoyiki" target="_blank" >
                <i class="icon icon-lg icon-github"></i>
                Github
              </a>
            </li>
        
            <li class="waves-block waves-effect">
              <a href="/about"  >
                <i class="icon icon-lg icon-user-circle"></i>
                About
              </a>
            </li>
        
      </ul>
    </div>
  </div>
</aside>

    <main id="main">
        <header class="top-header" id="header">
    <div class="flex-row">
        <a href="javascript:;" class="header-icon waves-effect waves-circle waves-light on" id="menu-toggle">
          <i class="icon icon-lg icon-navicon"></i>
        </a>
        <div class="flex-col header-title ellipsis">Scrapy学习（四） 爬取微博数据</div>
        
        <div class="search-wrap" id="search-wrap">
            <a href="javascript:;" class="header-icon waves-effect waves-circle waves-light" id="back">
                <i class="icon icon-lg icon-chevron-left"></i>
            </a>
            <input type="text" id="key" class="search-input" autocomplete="off" placeholder="输入感兴趣的关键字">
            <a href="javascript:;" class="header-icon waves-effect waves-circle waves-light" id="search">
                <i class="icon icon-lg icon-search"></i>
            </a>
        </div>
        
        
        <a href="javascript:;" class="header-icon waves-effect waves-circle waves-light" id="menuShare">
            <i class="icon icon-lg icon-share-alt"></i>
        </a>
        
    </div>
</header>
<header class="content-header post-header">

    <div class="container fade-scale">
        <h1 class="title">Scrapy学习（四） 爬取微博数据</h1>
        <h5 class="subtitle">
            
                <time datetime="2017-02-13T06:48:00.000Z" itemprop="datePublished" class="page-time">
  2017-02-13
</time>


	<ul class="article-category-list"><li class="article-category-list-item"><a class="article-category-list-link" href="/categories/Python/">Python</a></li></ul>

            
        </h5>
    </div>

    

</header>


<div class="container body-wrap">
    
    <aside class="post-widget">
        <nav class="post-toc-wrap" id="post-toc">
            <h4>TOC</h4>
            <ol class="post-toc"><li class="post-toc-item post-toc-level-2"><a class="post-toc-link" href="#前言"><span class="post-toc-number">1.</span> <span class="post-toc-text">前言</span></a></li><li class="post-toc-item post-toc-level-2"><a class="post-toc-link" href="#准备"><span class="post-toc-number">2.</span> <span class="post-toc-text">准备</span></a></li><li class="post-toc-item post-toc-level-2"><a class="post-toc-link" href="#创建Items"><span class="post-toc-number">3.</span> <span class="post-toc-text">创建Items</span></a></li><li class="post-toc-item post-toc-level-2"><a class="post-toc-link" href="#编写Spider"><span class="post-toc-number">4.</span> <span class="post-toc-text">编写Spider</span></a><ol class="post-toc-child"><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#start-url"><span class="post-toc-number">4.1.</span> <span class="post-toc-text">start_url</span></a></li><li class="post-toc-item post-toc-level-3"><a class="post-toc-link" href="#CloseSpider"><span class="post-toc-number">4.2.</span> <span class="post-toc-text">CloseSpider</span></a></li></ol></li><li class="post-toc-item post-toc-level-2"><a class="post-toc-link" href="#编写middlewares"><span class="post-toc-number">5.</span> <span class="post-toc-text">编写middlewares</span></a></li><li class="post-toc-item post-toc-level-2"><a class="post-toc-link" href="#编写cookie的获取方法"><span class="post-toc-number">6.</span> <span class="post-toc-text">编写cookie的获取方法</span></a></li><li class="post-toc-item post-toc-level-2"><a class="post-toc-link" href="#编写pipelines"><span class="post-toc-number">7.</span> <span class="post-toc-text">编写pipelines</span></a></li><li class="post-toc-item post-toc-level-2"><a class="post-toc-link" href="#总结"><span class="post-toc-number">8.</span> <span class="post-toc-text">总结</span></a></li></ol>
        </nav>
    </aside>
    
<article id="post-Scrapy学习（四）-爬取微博数据"
  class="post-article article-type-post fade" itemprop="blogPost">

    <div class="post-card">
        <h1 class="post-card-title">Scrapy学习（四） 爬取微博数据</h1>
        <div class="post-meta">
            <time class="post-time" title="2017年02月13日 14:48" datetime="2017-02-13T06:48:00.000Z"  itemprop="datePublished">2017-02-13</time>

            
	<ul class="article-category-list"><li class="article-category-list-item"><a class="article-category-list-link" href="/categories/Python/">Python</a></li></ul>



            
<span id="busuanzi_container_page_pv" title="文章总阅读量" style='display:none'>
    <i class="icon icon-eye icon-pr"></i><span id="busuanzi_value_page_pv"></span>
</span>


            

        </div>
        <div class="post-content" id="post-content" itemprop="postContent">
            <h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><blockquote>
<p><a href="https://segmentfault.com/a/1190000008253696?_ea=1616863" target="_blank" rel="external">Scrapy学习（三） 爬取豆瓣图书信息</a></p>
</blockquote>
<p>接上篇之后。这次来爬取需要登录才能访问的微博。<br>爬虫目标是获取用户的微博数、关注数、粉丝数。为建立用户关系图(尚未实现)做数据储备<br><a id="more"></a></p>
<h2 id="准备"><a href="#准备" class="headerlink" title="准备"></a>准备</h2><ul>
<li>安装第三方库<code>requests</code>和<code>pymongo</code></li>
<li>安装MongoDB</li>
<li>创建一个weibo爬虫项目</li>
</ul>
<p>如何创建Scrapy项目之前文章都已经提到了，直接进入主题。</p>
<h2 id="创建Items"><a href="#创建Items" class="headerlink" title="创建Items"></a>创建Items</h2><p>Item数据这部分我只需要个人信息，微博数，关注数、分数数这些基本信息就行。<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">ProfileItem</span><span class="params">(Item)</span>:</span></span><br><span class="line">    <span class="string">"""</span><br><span class="line">    账号的微博数、关注数、粉丝数及详情</span><br><span class="line">    """</span></span><br><span class="line">    _id = Field()</span><br><span class="line">    nick_name = Field()</span><br><span class="line">    profile_pic = Field()</span><br><span class="line">    tweet_stats = Field()</span><br><span class="line">    following_stats = Field()</span><br><span class="line">    follower_stats = Field()</span><br><span class="line">    sex = Field()</span><br><span class="line">    location = Field()</span><br><span class="line">    birthday = Field()</span><br><span class="line">    bio = Field()</span><br><span class="line">    </span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">FollowingItem</span><span class="params">(Item)</span>:</span></span><br><span class="line">    <span class="string">"""</span><br><span class="line">    关注的微博账号</span><br><span class="line">    """</span></span><br><span class="line">    _id = Field()</span><br><span class="line">    relationship = Field()</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">FollowedItem</span><span class="params">(Item)</span>:</span></span><br><span class="line">    <span class="string">"""</span><br><span class="line">    粉丝的微博账号</span><br><span class="line">    """</span></span><br><span class="line">    _id = Field()</span><br><span class="line">    relationship = Field()</span><br></pre></td></tr></table></figure></p>
<h2 id="编写Spider"><a href="#编写Spider" class="headerlink" title="编写Spider"></a>编写Spider</h2><p>为了方便爬虫，我们选择登陆的入口是手机版的微博<code>http://weibo.cn/</code>。<br><figure class="image-bubble">
                <div class="img-lightbox">
                    <div class="overlay"></div>
                    <img src="http://oi6538cys.bkt.clouddn.com/snipaste20170213_132630.png" alt="" title="">
                </div>
                <div class="image-caption"></div>
            </figure><br>其中微博的uid可以通过访问用户资料页或者从关注的<code>href</code>属性中获取</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">WeiboSpiderSpider</span><span class="params">(scrapy.Spider)</span>:</span></span><br><span class="line">    name = <span class="string">"weibo_spider"</span></span><br><span class="line">    allowed_domains = [<span class="string">"weibo.cn"</span>]</span><br><span class="line">    url = <span class="string">"http://weibo.cn/"</span></span><br><span class="line">    start_urls = [<span class="string">'2634877355'</span>,...] <span class="comment"># 爬取入口微博ID</span></span><br><span class="line">    task_set = set(start_urls) <span class="comment"># 待爬取集合</span></span><br><span class="line">    tasked_set = set() <span class="comment"># 已爬取集合</span></span><br><span class="line">    ...   </span><br><span class="line">    </span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">start_requests</span><span class="params">(self)</span>:</span></span><br><span class="line">        <span class="keyword">while</span> len(self.task_set) &gt; <span class="number">0</span> :</span><br><span class="line">            _id = self.task_set.pop()</span><br><span class="line">            <span class="keyword">if</span> _id <span class="keyword">in</span> self.tasked_set:</span><br><span class="line">                <span class="keyword">raise</span> CloseSpider(reason=<span class="string">"已存在该数据 %s "</span>% (_id) )</span><br><span class="line">            self.tasked_set.add(_id)</span><br><span class="line">            info_url = self.url + _id</span><br><span class="line">            info_item = ProfileItem()</span><br><span class="line">            following_url = info_url + <span class="string">"/follow"</span></span><br><span class="line">            following_item = FollowingItem()</span><br><span class="line">            following_item[<span class="string">"_id"</span>] = _id</span><br><span class="line">            following_item[<span class="string">"relationship"</span>] = []</span><br><span class="line">            follower_url = info_url + <span class="string">"/fans"</span></span><br><span class="line">            follower_item = FollowedItem()</span><br><span class="line">            follower_item[<span class="string">"_id"</span>] = _id</span><br><span class="line">            follower_item[<span class="string">"relationship"</span>] = []</span><br><span class="line">            <span class="keyword">yield</span> scrapy.Request(info_url, meta=&#123;<span class="string">"item"</span>:info_item&#125;, callback=self.account_parse)</span><br><span class="line">            <span class="keyword">yield</span> scrapy.Request(following_url, meta=&#123;<span class="string">"item"</span>:following_item&#125;, callback=self.relationship_parse)</span><br><span class="line">            <span class="keyword">yield</span> scrapy.Request(follower_url, meta=&#123;<span class="string">"item"</span>:follower_item&#125;, callback=self.relationship_parse)</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">account_parse</span><span class="params">(self, response)</span>:</span></span><br><span class="line">        item = response.meta[<span class="string">"item"</span>]</span><br><span class="line">        sel = scrapy.Selector(response)</span><br><span class="line">        profile_url = sel.xpath(<span class="string">"//div[@class='ut']/a/@href"</span>).extract()[<span class="number">1</span>]</span><br><span class="line">        counts = sel.xpath(<span class="string">"//div[@class='u']/div[@class='tip2']"</span>).extract_first()</span><br><span class="line">        item[<span class="string">'_id'</span>] = re.findall(<span class="string">u'^/(\d+)/info'</span>,profile_url)[<span class="number">0</span>]</span><br><span class="line">        item[<span class="string">'tweet_stats'</span>] = re.findall(<span class="string">u'微博\[(\d+)\]'</span>, counts)[<span class="number">0</span>]</span><br><span class="line">        item[<span class="string">'following_stats'</span>] = re.findall(<span class="string">u'关注\[(\d+)\]'</span>, counts)[<span class="number">0</span>]</span><br><span class="line">        item[<span class="string">'follower_stats'</span>] = re.findall(<span class="string">u'粉丝\[(\d+)\]'</span>, counts)[<span class="number">0</span>]</span><br><span class="line">        <span class="keyword">if</span> int(item[<span class="string">'tweet_stats'</span>]) &lt; <span class="number">4500</span> <span class="keyword">and</span> int(item[<span class="string">'following_stats'</span>]) &gt; <span class="number">1000</span> <span class="keyword">and</span> int(item[<span class="string">'follower_stats'</span>]) &lt; <span class="number">500</span>:</span><br><span class="line">            <span class="keyword">raise</span> CloseSpider(<span class="string">"僵尸粉"</span>)</span><br><span class="line">        <span class="keyword">yield</span> scrapy.Request(<span class="string">"http://weibo.cn"</span>+profile_url, meta=&#123;<span class="string">"item"</span>: item&#125;,callback=self.profile_parse)</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">profile_parse</span><span class="params">(self,response)</span>:</span></span><br><span class="line">        item = response.meta[<span class="string">'item'</span>]</span><br><span class="line">        sel = scrapy.Selector(response)</span><br><span class="line">        info = sel.xpath(<span class="string">"//div[@class='tip']/following-sibling::div[@class='c']"</span>).extract_first()</span><br><span class="line">        item[<span class="string">"profile_pic"</span>] = sel.xpath(<span class="string">"//div[@class='c']/img/@src"</span>).extract_first()</span><br><span class="line">        item[<span class="string">"nick_name"</span>] = re.findall(<span class="string">u'昵称:(.*?)&lt;br&gt;'</span>,info)[<span class="number">0</span>]</span><br><span class="line">        item[<span class="string">"sex"</span>] = re.findall(<span class="string">u'性别:(.*?)&lt;br&gt;'</span>,info) <span class="keyword">and</span> re.findall(<span class="string">u'性别:(.*?)&lt;br&gt;'</span>,info)[<span class="number">0</span>] <span class="keyword">or</span> <span class="string">''</span></span><br><span class="line">        item[<span class="string">"location"</span>] = re.findall(<span class="string">u'地区:(.*?)&lt;br&gt;'</span>,info) <span class="keyword">and</span> re.findall(<span class="string">u'地区:(.*?)&lt;br&gt;'</span>,info)[<span class="number">0</span>] <span class="keyword">or</span> <span class="string">''</span></span><br><span class="line">        item[<span class="string">"birthday"</span>] = re.findall(<span class="string">u'生日:(.*?)&lt;br&gt;'</span>,info) <span class="keyword">and</span> re.findall(<span class="string">u'生日:(.*?)&lt;br&gt;'</span>,info)[<span class="number">0</span>] <span class="keyword">or</span> <span class="string">''</span></span><br><span class="line">        item[<span class="string">"bio"</span>] = re.findall(<span class="string">u'简介:(.*?)&lt;br&gt;'</span>,info) <span class="keyword">and</span> re.findall(<span class="string">u'简介:(.*?)&lt;br&gt;'</span>,info)[<span class="number">0</span>] <span class="keyword">or</span> <span class="string">''</span></span><br><span class="line">        <span class="keyword">yield</span> item</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">relationship_parse</span><span class="params">(self, response)</span>:</span></span><br><span class="line">        item = response.meta[<span class="string">"item"</span>]</span><br><span class="line">        sel = scrapy.Selector(response)</span><br><span class="line">        uids = sel.xpath(<span class="string">"//table/tr/td[last()]/a[last()]/@href"</span>).extract()</span><br><span class="line">        new_uids = []</span><br><span class="line">        <span class="keyword">for</span> uid <span class="keyword">in</span> uids:</span><br><span class="line">            <span class="keyword">if</span> <span class="string">"uid"</span> <span class="keyword">in</span> uid:</span><br><span class="line">                new_uids.append(re.findall(<span class="string">'uid=(\d+)&amp;'</span>,uid)[<span class="number">0</span>])</span><br><span class="line">            <span class="keyword">else</span>:</span><br><span class="line">                <span class="keyword">try</span>:</span><br><span class="line">                    new_uids.append(re.findall(<span class="string">'/(\d+)'</span>, uid)[<span class="number">0</span>])</span><br><span class="line">                <span class="keyword">except</span>:</span><br><span class="line">                    print(<span class="string">'--------'</span>,uid)</span><br><span class="line">                    <span class="keyword">pass</span></span><br><span class="line">        item[<span class="string">"relationship"</span>].extend(new_uids)</span><br><span class="line">        <span class="keyword">for</span> i <span class="keyword">in</span> new_uids:</span><br><span class="line">            <span class="keyword">if</span> i <span class="keyword">not</span> <span class="keyword">in</span> self.tasked_set:</span><br><span class="line">                self.task_set.add(i)</span><br><span class="line">        next_page = sel.xpath(<span class="string">"//*[@id='pagelist']/form/div/a[text()='下页']/@href"</span>).extract_first()</span><br><span class="line">        <span class="keyword">if</span> next_page:</span><br><span class="line">            <span class="keyword">yield</span> scrapy.Request(<span class="string">"http://weibo.cn"</span>+next_page, meta=&#123;<span class="string">"item"</span>: item&#125;,callback=self.relationship_parse)</span><br><span class="line">        <span class="keyword">else</span>:</span><br><span class="line">            <span class="keyword">yield</span> item</span><br></pre></td></tr></table></figure>
<p>代码中值得注意的地方有几个。</p>
<h3 id="start-url"><a href="#start-url" class="headerlink" title="start_url"></a>start_url</h3><p>这里我们填写的是微博的uid，有的用户有自定义域名（如上图），要访问后才能得到真正的uid<br><code>start_url</code> 填写的初始种子数要在10个以上。这是为了确保后面我们爬取到的新的种子能够加入到待爬取的队列中。10个以上的规定是从<a href="https://doc.scrapy.org/en/latest/topics/settings.html#reactor-threadpool-maxsize" target="_blank" rel="external">Scrapy文档</a>中查得的</p>
<blockquote>
<p>####REACTOR_THREADPOOL_MAXSIZE###<br>Default: <code>10</code><br>线程数是<code>Twisted</code>线程池的默认大小(The maximum limit for Twisted Reactor thread pool size.)</p>
</blockquote>
<h3 id="CloseSpider"><a href="#CloseSpider" class="headerlink" title="CloseSpider"></a>CloseSpider</h3><p>当遇到不需要的继续爬取的连接时(如已经爬取过的链接，定义的僵尸粉链接等等)，就可以用<a href="https://doc.scrapy.org/en/latest/topics/exceptions.html#closespider" target="_blank" rel="external">CloseSpider</a>关闭当前爬虫线程</p>
<h2 id="编写middlewares"><a href="#编写middlewares" class="headerlink" title="编写middlewares"></a>编写middlewares</h2><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">CookiesMiddleware</span><span class="params">(object)</span>:</span></span><br><span class="line">    <span class="string">""" 换Cookie """</span></span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">process_request</span><span class="params">(self, request, spider)</span>:</span></span><br><span class="line">        cookie = random.choice(cookies)</span><br><span class="line">        request.cookies = cookie</span><br></pre></td></tr></table></figure>
<h2 id="编写cookie的获取方法"><a href="#编写cookie的获取方法" class="headerlink" title="编写cookie的获取方法"></a>编写cookie的获取方法</h2><p>这里我原本是想用手机版的微博去模拟登陆的，奈何验证码是在是太难搞了。所以我直接用网上有人编写好的登陆网页版微博的代码<a href="https://github.com/LiuXingMing/SinaSpider/blob/master/Sina_spider1/Sina_spider1/cookies.py" target="_blank" rel="external">SinaSpider</a> 这位写的很好，有兴趣的可以去看看。其中还有另一位写了<a href="https://github.com/xchaoinfo/fuck-login/blob/master/007%20weibo.com/weibo.com.py" target="_blank" rel="external">模拟登陆（带验证码）</a> 经测试可用。只不过我还没想好怎么嵌入到我的项目中。<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># encoding=utf-8</span></span><br><span class="line"><span class="keyword">import</span> json</span><br><span class="line"><span class="keyword">import</span> base64</span><br><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"></span><br><span class="line">myWeiBo = [</span><br><span class="line">    &#123;<span class="string">'no'</span>: <span class="string">'xx@sina.com'</span>, <span class="string">'psw'</span>: <span class="string">'xx'</span>&#125;,</span><br><span class="line">    &#123;<span class="string">'no'</span>: <span class="string">'xx@qq.com'</span>, <span class="string">'psw'</span>: <span class="string">'xx'</span>&#125;,</span><br><span class="line">]</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">getCookies</span><span class="params">(weibo)</span>:</span></span><br><span class="line">    <span class="string">""" 获取Cookies """</span></span><br><span class="line">    cookies = []</span><br><span class="line">    loginURL = <span class="string">r'https://login.sina.com.cn/sso/login.php?client=ssologin.js(v1.4.15)'</span></span><br><span class="line">    <span class="keyword">for</span> elem <span class="keyword">in</span> weibo:</span><br><span class="line">        account = elem[<span class="string">'no'</span>]</span><br><span class="line">        password = elem[<span class="string">'psw'</span>]</span><br><span class="line">        username = base64.b64encode(account.encode(<span class="string">'utf-8'</span>)).decode(<span class="string">'utf-8'</span>)</span><br><span class="line">        postData = &#123;</span><br><span class="line">            <span class="string">"entry"</span>: <span class="string">"sso"</span>,</span><br><span class="line">            <span class="string">"gateway"</span>: <span class="string">"1"</span>,</span><br><span class="line">            <span class="string">"from"</span>: <span class="string">"null"</span>,</span><br><span class="line">            <span class="string">"savestate"</span>: <span class="string">"30"</span>,</span><br><span class="line">            <span class="string">"useticket"</span>: <span class="string">"0"</span>,</span><br><span class="line">            <span class="string">"pagerefer"</span>: <span class="string">""</span>,</span><br><span class="line">            <span class="string">"vsnf"</span>: <span class="string">"1"</span>,</span><br><span class="line">            <span class="string">"su"</span>: username,</span><br><span class="line">            <span class="string">"service"</span>: <span class="string">"sso"</span>,</span><br><span class="line">            <span class="string">"sp"</span>: password,</span><br><span class="line">            <span class="string">"sr"</span>: <span class="string">"1440*900"</span>,</span><br><span class="line">            <span class="string">"encoding"</span>: <span class="string">"UTF-8"</span>,</span><br><span class="line">            <span class="string">"cdult"</span>: <span class="string">"3"</span>,</span><br><span class="line">            <span class="string">"domain"</span>: <span class="string">"sina.com.cn"</span>,</span><br><span class="line">            <span class="string">"prelt"</span>: <span class="string">"0"</span>,</span><br><span class="line">            <span class="string">"returntype"</span>: <span class="string">"TEXT"</span>,</span><br><span class="line">        &#125;</span><br><span class="line">        session = requests.Session()</span><br><span class="line">        r = session.post(loginURL, data=postData)</span><br><span class="line">        jsonStr = r.content.decode(<span class="string">'gbk'</span>)</span><br><span class="line">        info = json.loads(jsonStr)</span><br><span class="line">        <span class="keyword">if</span> info[<span class="string">"retcode"</span>] == <span class="string">"0"</span>:</span><br><span class="line">            print(<span class="string">"Get Cookie Success!( Account:%s )"</span> % account)</span><br><span class="line">            cookie = session.cookies.get_dict()</span><br><span class="line">            cookies.append(cookie)</span><br><span class="line">        <span class="keyword">else</span>:</span><br><span class="line">            print(<span class="string">"Failed!( Reason:%s )"</span> % info[<span class="string">"reason"</span>].encode(<span class="string">"utf-8"</span>))</span><br><span class="line">    <span class="keyword">return</span> cookies</span><br><span class="line"></span><br><span class="line">cookies = getCookies(myWeiBo)</span><br></pre></td></tr></table></figure></p>
<p>登陆-反爬虫的这部分应该是整个项目中最难的地方了。<del>好多地方我都还不太懂。以后有空在研究</del></p>
<h2 id="编写pipelines"><a href="#编写pipelines" class="headerlink" title="编写pipelines"></a>编写pipelines</h2><p>这边只需要主要什么类型的Item存到那张表里就行了<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">MongoDBPipeline</span><span class="params">(object)</span>:</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span><span class="params">(self)</span>:</span></span><br><span class="line">        connection = MongoClient(</span><br><span class="line">            host=settings[<span class="string">'MONGODB_SERVER'</span>],</span><br><span class="line">            port=settings[<span class="string">'MONGODB_PORT'</span>]</span><br><span class="line">        )</span><br><span class="line">        db = connection[settings[<span class="string">'MONGODB_DB'</span>]]</span><br><span class="line">        self.info = db[settings[<span class="string">'INFO'</span>]]</span><br><span class="line">        self.following = db[settings[<span class="string">'FOLLOWING'</span>]]</span><br><span class="line">        self.followed = db[settings[<span class="string">'FOLLOWED'</span>]]</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">process_item</span><span class="params">(self, item, spider)</span>:</span></span><br><span class="line"></span><br><span class="line">        <span class="keyword">if</span> isinstance(item, ProfileItem):</span><br><span class="line">            self.info.insert(dict(item))</span><br><span class="line">        <span class="keyword">elif</span> isinstance(item, FollowingItem):</span><br><span class="line">            self.following.insert(dict(item))</span><br><span class="line">        <span class="keyword">elif</span> isinstance(item, FollowedItem):</span><br><span class="line">            self.followed.insert(dict(item))</span><br><span class="line">        log.msg(<span class="string">"Weibo  added to MongoDB database!"</span>,</span><br><span class="line">                level=log.DEBUG, spider=spider)</span><br><span class="line">        <span class="keyword">return</span> item</span><br></pre></td></tr></table></figure></p>
<p>运行一下程序，就能看到MongoDB中有了我们要的数据了<br><figure class="image-bubble">
                <div class="img-lightbox">
                    <div class="overlay"></div>
                    <img src="http://oi6538cys.bkt.clouddn.com/snipaste20170213_141218.png" alt="" title="">
                </div>
                <div class="image-caption"></div>
            </figure></p>
<h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><ul>
<li><code>settings</code>中的<code>DOWNLOAD_DELAY</code>设置5才能防止被微博BAN掉</li>
<li>尝试在利用cookies登陆失败时使用模拟登陆，但是效果很不理想</li>
<li>尝试用代理IP池反爬虫，但是尝试失败<del>主要是不太会</del></li>
<li>未来将利用D3.js将爬到的数据绘制出来(<del>或许吧</del>)</li>
</ul>
<p>项目地址：<a href="https://github.com/amoyiki/weibo_spider" target="_blank" rel="external">weibo_spider</a></p>

        </div>

        <blockquote class="post-copyright">
    <div class="content">
        
<span class="post-time">
    最后更新时间：<time datetime="2017-02-13T06:50:38.033Z" itemprop="dateUpdated">2017年2月13日 14:50</time>
</span><br>


        这里写留言或版权声明：<a href="/2017/02/13/Scrapy学习（四）-爬取微博数据/" target="_blank" rel="external">http://amoyiki.com/2017/02/13/Scrapy学习（四）-爬取微博数据/</a>
    </div>
    <footer>
        <a href="http://amoyiki.com">
            <img src="/images/avatar.jpg" alt="amoyiki">
            amoyiki
        </a>
    </footer>
</blockquote>

        
<div class="page-reward">
    <a id="rewardBtn" href="javascript:;" class="page-reward-btn waves-effect waves-circle waves-light">赏</a>
</div>



        <div class="post-footer">
            
	<ul class="article-tag-list"><li class="article-tag-list-item"><a class="article-tag-list-link" href="/tags/Scrapy/">Scrapy</a></li><li class="article-tag-list-item"><a class="article-tag-list-link" href="/tags/学习/">学习</a></li></ul>


            
<div class="page-share-wrap">
    

<div class="page-share" id="pageShare">
    <ul class="reset share-icons">
      <li>
        <a class="weibo share-sns" target="_blank" href="http://service.weibo.com/share/share.php?url=http://amoyiki.com/2017/02/13/Scrapy学习（四）-爬取微博数据/&title=《Scrapy学习（四） 爬取微博数据》 — 四畳半神话大系&pic=http://amoyiki.com/images/avatar.jpg" data-title="微博">
          <i class="icon icon-weibo"></i>
        </a>
      </li>
      <li>
        <a class="weixin share-sns wxFab" href="javascript:;" data-title="微信">
          <i class="icon icon-weixin"></i>
        </a>
      </li>
      <li>
        <a class="qq share-sns" target="_blank" href="http://connect.qq.com/widget/shareqq/index.html?url=http://amoyiki.com/2017/02/13/Scrapy学习（四）-爬取微博数据/&title=《Scrapy学习（四） 爬取微博数据》 — 四畳半神话大系&source=前言
Scrapy学习（三） 爬取豆瓣图书信息

接上篇之后。这次来爬取需要登录才能访问的微博。爬虫目标是获取用户的微博数、关注数、粉丝数。为建立用户关系..." data-title=" QQ">
          <i class="icon icon-qq"></i>
        </a>
      </li>
      <li>
        <a class="facebook share-sns" target="_blank" href="https://www.facebook.com/sharer/sharer.php?u=http://amoyiki.com/2017/02/13/Scrapy学习（四）-爬取微博数据/" data-title=" Facebook">
          <i class="icon icon-facebook"></i>
        </a>
      </li>
      <li>
        <a class="twitter share-sns" target="_blank" href="https://twitter.com/intent/tweet?text=《Scrapy学习（四） 爬取微博数据》 — 四畳半神话大系&url=http://amoyiki.com/2017/02/13/Scrapy学习（四）-爬取微博数据/&via=http://amoyiki.com" data-title=" Twitter">
          <i class="icon icon-twitter"></i>
        </a>
      </li>
      <li>
        <a class="google share-sns" target="_blank" href="https://plus.google.com/share?url=http://amoyiki.com/2017/02/13/Scrapy学习（四）-爬取微博数据/" data-title=" Google+">
          <i class="icon icon-google-plus"></i>
        </a>
      </li>
    </ul>
 </div>



    <a href="javascript:;" id="shareFab" class="page-share-fab waves-effect waves-circle">
        <i class="icon icon-share-alt icon-lg"></i>
    </a>
</div>



        </div>
    </div>

    
<nav class="post-nav flex-row flex-justify-between">
  
    <div class="waves-block waves-effect prev">
      <a href="/2017/03/20/CareerCup-1-4-Replace-Spaces-替换空格/" id="post-prev" class="post-nav-link">
        <div class="tips"><i class="icon icon-angle-left icon-lg icon-pr"></i> Prev</div>
        <h4 class="title">[CareerCup] 1.4 Replace Spaces</h4>
      </a>
    </div>
  

  
    <div class="waves-block waves-effect next">
      <a href="/2017/02/04/Scrapy学习（三）-爬取豆瓣图书信息/" id="post-next" class="post-nav-link">
        <div class="tips">Next <i class="icon icon-angle-right icon-lg icon-pl"></i></div>
        <h4 class="title">Scrapy学习（三） 爬取豆瓣图书信息</h4>
      </a>
    </div>
  
</nav>



    




<section class="comments" id="comments">
    <div id="disqus_thread"></div>
    <script>
    var disqus_shortname = 'amoyiki';
    lazyScripts.push('//' + disqus_shortname + '.disqus.com/embed.js')
    </script>
    <noscript>Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>
</section>




</article>

<div id="reward" class="page-modal reward-lay">
    <a class="close" href="javascript:;"><i class="icon icon-close"></i></a>
    <h3 class="reward-title">
        <i class="icon icon-quote-left"></i>
        谢谢大爷~
        <i class="icon icon-quote-right"></i>
    </h3>
    <ul class="reward-items">
        

        
    </ul>
</div>



</div>

        <footer class="footer">
    <div class="top">
        
<p>
    <span id="busuanzi_container_site_uv" style='display:none'>
        站点总访客数：<span id="busuanzi_value_site_uv"></span>
    </span>
    <span id="busuanzi_container_site_pv" style='display:none'>
        站点总访问量：<span id="busuanzi_value_site_pv"></span>
    </span>
</p>


        <p>
            <span><a href="/atom.xml" target="_blank" class="rss" title="rss"><i class="icon icon-lg icon-rss"></i></a></span>
            <span>博客内容遵循 <a href="http://creativecommons.org/licenses/by-nc-sa/4.0/" target="_blank">知识共享 署名 - 非商业性 - 相同方式共享 4.0协议</a></span>
        </p>
    </div>
    <div class="bottom">
        <p>
            <span>Power by <a href="http://hexo.io/" target="_blank">Hexo</a> Theme <a href="https://github.com/yscoder/hexo-theme-indigo" target="_blank">indigo</a></span>
            <span>四畳半神话大系 &copy; 2015 - 2017</span>
        </p>
    </div>
</footer>

    </main>
    <div class="mask" id="mask"></div>
<a href="javascript:;" id="gotop" class="waves-effect waves-circle waves-light"><span class="icon icon-lg icon-chevron-up"></span></a>



<div class="global-share" id="globalShare">
    <ul class="reset share-icons">
      <li>
        <a class="weibo share-sns" target="_blank" href="http://service.weibo.com/share/share.php?url=http://amoyiki.com/2017/02/13/Scrapy学习（四）-爬取微博数据/&title=《Scrapy学习（四） 爬取微博数据》 — 四畳半神话大系&pic=http://amoyiki.com/images/avatar.jpg" data-title="微博">
          <i class="icon icon-weibo"></i>
        </a>
      </li>
      <li>
        <a class="weixin share-sns wxFab" href="javascript:;" data-title="微信">
          <i class="icon icon-weixin"></i>
        </a>
      </li>
      <li>
        <a class="qq share-sns" target="_blank" href="http://connect.qq.com/widget/shareqq/index.html?url=http://amoyiki.com/2017/02/13/Scrapy学习（四）-爬取微博数据/&title=《Scrapy学习（四） 爬取微博数据》 — 四畳半神话大系&source=前言
Scrapy学习（三） 爬取豆瓣图书信息

接上篇之后。这次来爬取需要登录才能访问的微博。爬虫目标是获取用户的微博数、关注数、粉丝数。为建立用户关系..." data-title=" QQ">
          <i class="icon icon-qq"></i>
        </a>
      </li>
      <li>
        <a class="facebook share-sns" target="_blank" href="https://www.facebook.com/sharer/sharer.php?u=http://amoyiki.com/2017/02/13/Scrapy学习（四）-爬取微博数据/" data-title=" Facebook">
          <i class="icon icon-facebook"></i>
        </a>
      </li>
      <li>
        <a class="twitter share-sns" target="_blank" href="https://twitter.com/intent/tweet?text=《Scrapy学习（四） 爬取微博数据》 — 四畳半神话大系&url=http://amoyiki.com/2017/02/13/Scrapy学习（四）-爬取微博数据/&via=http://amoyiki.com" data-title=" Twitter">
          <i class="icon icon-twitter"></i>
        </a>
      </li>
      <li>
        <a class="google share-sns" target="_blank" href="https://plus.google.com/share?url=http://amoyiki.com/2017/02/13/Scrapy学习（四）-爬取微博数据/" data-title=" Google+">
          <i class="icon icon-google-plus"></i>
        </a>
      </li>
    </ul>
 </div>


<div class="page-modal wx-share" id="wxShare">
    <a class="close" href="javascript:;"><i class="icon icon-close"></i></a>
    <p>扫一扫，分享到微信</p>
    <img src="" alt="微信分享二维码">
</div>




    <script src="//cdn.bootcss.com/node-waves/0.7.4/waves.min.js"></script>
<script>
var BLOG = { ROOT: '/', SHARE: true, REWARD: true };



</script>

<script src="/js/main.min.js?v=1.4.14"></script>


<div class="search-panel" id="search-panel">
    <ul class="search-result" id="search-result"></ul>
</div>
<template id="search-tpl">
<li class="item">
    <a href="{path}" class="waves-block waves-effect">
        <div class="title ellipsis" title="{title}">{title}</div>
        <div class="flex-row flex-middle">
            <div class="tags ellipsis">
                {tags}
            </div>
            <time class="flex-col time">{date}</time>
        </div>
    </a>
</li>
</template>

<script src="/js/search.min.js?v=1.4.14" async></script>






<script async src="//dn-lbstatics.qbox.me/busuanzi/2.3/busuanzi.pure.mini.js"></script>


</body>
</html>
